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Remarks 

Claims 2-6 are pending in the application. The rejections of the claims of the August 23 , 
2004, Office Action have been withdrawn, as a result of Applicant's submission of claim 
amendment and remarks associated with the Request for Continued Examination. A new ground 
of rejection, however, is made in the instant Office Action in view of newly found prior art 
references. 

A. Objection to the Specification 

The instant Action objected to the specification because of a typographical error at page 
6, line 26. Applicant has reviewed the specification and found one additional typographical error, 
at page 8, line 13. Both errors are corrected by the amendments to the specification above, which 
introduce no new matter into the application. 

B. Claim rejections under 35 U.S.C. §103 

(a) Claims 2-6 were rejected under 35 U.S.C. § 103(a) as being obvious in light of the combined 
teachings of Dumoulin 1 and Funda 2 . Applicants respectfully disagree and offer the following 
remarks to clarify the patentable differences between the presently claimed invention and the cited 
art. 

(b) With respect to claim 2, some text may be missing from the Action, as item 5 of page 3 
begins with what appears to be two cut-off paragraphs, "...the stereo display system 
comprising:...", and "The synchronization processes are defined by..." It appears that the latter 
partial paragraph is completed three paragraphs later (also on page 3 of the Action), however the 
former is not. Applicants request that if additional text was intended by the Examiner to be 
included in the Action that would require a response from Applicants beyond the response that 



1 U.S. Pat. No. 5,526,812 issued 18 June 1996 to Dumouin, et al. 

2 U.S. Pat. No. 5,749,362 issued 12 May 1998 to Funda, et al. 
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appears below, that Applicants be provided an opportunity to supplement the present response 
without incurring any additional fees. 

(c) Claim 2 recites: 

Method of creating a stereoscopic haptic virtual environment, comprising the steps of: 
generating stereoscopic graphics and haptic scene components; 
synchronizing the stereoscopic graphics and haptic scene components; and 
presenting the synchronized stereoscopic graphics and haptic scene components to a user. 

The Action asserts that Dumoulin teaches all the subject matter of claim 2, except for the 
use of an input device instead of a haptic scene input device. More particularly, it is stated that 
in both Dumoulin embodiments, the computer image must be registered (coincide) with the 
external structures as viewed by operator 350, and that initialization may be accomplished by 
manual input from the operator to rotate, translate and scale the computer generated image(s) 
until they coincide with the scene observed through the semi-transparent screen, or by employing 
tracking device 50 to set initial parameters. Further, it is asserted that once the 3D model and the 
visual image of patient 1 are aligned, tracking device 50 keeps the view angles and field of view 
consistent, allowing real-time interactive synchronization between the operator's view of patient 
1 and the computer generated image(s) defined by a stereoscopic viewer (252) (fig. 2, 
stereoscopic graphics) and the workstation view input device (60) (fig. 2, scene components.) 

Applicants respectfully submit that the references do not teach or suggest the presently 
claimed invention as alleged in the Action. The synchronizing process addressed by the 
references involves aligning the mono or stereoscopic rendered image of the computer generated 
3D model of the internal structures with the visual image of patient 1 (i.e. the observable real 
external structures) through the semi-transparent screen. A tracking device is used to maintain the 
real-time interactive synchronization between the two, by keeping track of the view angle and the 
field of view. The synchronization processes taught by the cited references and the presently 
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claimed invention are different mechanisms. The synchronizing stereoscopic graphics and haptic 
scene components limitation recited in claim 2 ensures that the optical and haptic presentations of 
3D models are consistent and synchronized (see instant specification, at page 8, lines 3-6.) The 
synchronization of a graphics environment running at, for example, 30 Hz, and the hap tics 
environment executing at, for example, 1,000 Hz, achieves an effective integrated virtual 
environment. 

The haptic virtual environment of the presently claimed invention may be created 
independently of its graphics representation (such as described in Applicants' paper 
"Development of Haptic Stereoscopic Virtual Environments", June 1999 Proc. 12 th Symp. 
IEEE/Computer-Based Medical Systems CBMS.) The virtual reality integration process that 
combines graphics and haptics also requires collision detection and force feedback computation 
algorithms, as shown in Figure 1 of the instant patent application. Collision detection and force 
feedback computation algorithms are essential for all haptic applications providing a realistic 
sense of touch for virtual objects. 

Neither Dumoulin nor Funda teach creating a haptic virtual environment or computing 
collision detection or force feedback. A distinction is to be drawn between references to haptics in 
the context of providing a calibrated sense of touch, via a haptic device, that is felt by a user 
when navigating a virtual environment, versus providing vibrational feedback when two objects 
are placed in alignment. Funda's system relates to the latter application where joystick vibration 
is used to confirm alignment. The Action's citation to a "tactile" input device (col. 16, lines 53- 
65), refers to such a signal produced when graphical object or a surgical instrument is properly 
placed in the vicinity of the current anatomical feature of interest. (Funda, col. 14, line 1 through 
col. 16, line 52) In contrast, the haptic device of the presently claimed invention refers to a device 
providing feedback that is proportional to the properties of virtual objects being modeled. This 
requires an additional software feedback loop that must be synchronized with the graphics (either 
mono or stereoscopic). This approach makes it possible to differentiate tissue properties of 
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computer generated objects and that feature makes the additional complexity of the software 
worthwhile. Even in combination, Dumoulin and Funda fail to teach feeling the objects existing in 
a virtual environment, despite their disclosures of approaches, distinct from those of the 
presently claimed invention (wherein an OpenGl API is used with astencil buffer to generate the 
stereoscopic 3D displays), of generating computer based stereoscopic image based displays. 

Dumoulin, contrary to the assertion in the Action, does not teach generating stereoscopic 
graphics and haptic scene components as recited in claim 2. Rather, Dumoulin discloses creation 
of 3D objects represented as images (col. 2, lines 36, 59, 64; col. 4, lines 42, 58, 66; col. 5, lines 
42, 53; col. 6, lines 1, 7, 10, 14, 18-21, 27-28, 42.) Representation of 3D models by RGB values 
(col. 4, lines 4-18, 50) is useful only in displays. Such representations do not include the 3D 
information, the geometry, needed to generate haptic representation of the virtual environment. 
Two separate software structures (one for the stereoscopic graphics scene and another for the 
haptic scene) consisting of hierarchical data, referred to as scene components in Applicants' 
specification, are required along with the collision detection and force feedback computation 
mechanisms in order to create an interactive haptic virtual environment. 

In the presently claimed invention, haptics and graphics components are represented in a 
computer using information such as geometry, position, orientation, visual properties (e.g., color 
and image texture), and haptic properties (e.g., stiffness, damping, static friction, dynamic 
friction). These scene components are organized into a hierarchical construct (scene graph) that is 
used for graphics and haptics rendering. 

Funda fails to teach or suggest the claimed elements not described by Dumoulin. Funda 
discloses that it is possible to interface to a haptic input device. Creating a haptic application, 
even after a method of interfacing to a haptic device is known, is non-obvious, as it becomes 
important to synchronize the graphic and haptic software loops, as discussed above. This critical 
point is not addressed by Funda. 
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The Action asserts that "the haptic scene input device/haptic scene components are 
defined by detection of virtual object and determination/application of the tactile feedback", 
however neither Dumoulin nor Funda use a concept of haptic scene. Funda addresses tactile 
feedback in a paragraph at col. 16, lines 53-67, but not haptic feedback (with the differences 
discussed above). Funda (col. 16, lines 59-63) describes equipping a joystick control with a 
computer controlled vibrator: "The tactile feedback can be delivered to the surgeon's hand or 
finger (whichever is in contact with the joystick) by instrumenting the joystick control with a 
computer controlled vibrator." Incorporation of such a device would require only a simple 
software modification since there is no feedback loop to synchronize with the graphics (as 
discussed above). This is clearly distinct from the haptic scene and haptic scene components and 
mechanisms of the presently claimed invention. For example the 3-D images viewed through 
Funda's goggles do not define graphics scene component. This simply provides a 3-D visual 
display. What is missing is the 3D geometry needed for a graphic scene component as described 
in the instant application (page 5, lines 17-20, and Figure 1). Graphics scene components simply 
cannot be generated with Funda's approach, as asserted in the Action. 

This contention is confirmed by careful reading of Funda (col. 5, line 64 - col. 6, linel2): 
"In one preferred embodiment, the stereo display system is a StereoGraphics CrystalEyes 
(trademark of StereoGraphics, Inc.) system, where the two video signals are displayed on a 
stereoscopic monitor which alternatively displays the left and right eye image at a frequency of 
120 Hz, updating the video information for each eye 60 times per second. The surgeon wears 
stereoscopic liquid crystal (LC) goggles 273, which are synchronized with the monitor and 
alternatively block light from entering left and right eye such that the left eye receives only the 
video signal from the left camera and the right eye receives only the information from the right 
camera. The frequency of alternation between left and right images is sufficiently high such that 
the surgeon perceives no flicker but rather a continuous stereoscopic image of the patient's 
anatomy. Other stereo display technologies arc available and may be used." This clearly defines 
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an analog display without the use of any computers or any means to generate graphic scene 
components. 

Similarly, the method of relaying non- visual information to the surgeon, by tactile 
feedback, does not imply that the virtual object is detected. This actually has to do with 
synchronizing images of the external and internal objects and making sure that the computer 
generated image of the internal object is properly placed, manually by the operator, over the 
external object. In Funda (Summary of Invention, col. 3, line 59-col. 4, line 2), the internal object 
in question is designated by an operator with the visual means, and then placed over the external 
object. Many scenarios and methods are discussed in Funda for computing the single specific 
location of the designated virtual object (see col. 4, line 3 1- col. 5, line 41, col. 5, lines 48-63, col7, 
lines 23-67, col. 8, lines 1 1-43, col. 8, line 53 - col. 11, line 25, col. 13, lines 14-63.) Placing a 
virtual object and computing its position is completely different from navigating through a 3D 
virtual space, using collision detection to find the virtual objects and then computing forces to be 
applied at each point of collision. Only in the latter process (the claimed invention) can different 
objects be felt differently. 

(d) In light of the foregoing, Applicants respectfully submit that independent claim 2 is 
nonobvious in light of the cited art. Claims 3-6 depend from independent claim 2, and if an 
independent claim is nonobvious, claims that depend from the independent claim are also 
nonobvious. In re Fine, 837 F.2d 1071, 1071; 5 USPQ2d 1596 (Fed. Cir. 1988). Further 
comments follow, however, responsive to additional points made in the instant Action regarding 
the dependent claims. 

(e) With regard to claim 3, the instant Action cites to Funda's joystick input device. The 
presently claimed invention encompasses a wide variety of input devices. In one non-limiting 
embodiment described in the instant specification, the visual cue comprises a sphere representing 
the position of the haptic device (provided by a 12 bit optical encoder that is an integral part of 
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the haptic device.) The sphere changes color from white to green when the device collides with 
any part of any virtual object (defined by the haptic scene graph of claim 2.) At each point of 
collision, the operator feels the hardness and the texture of the object at that point. The hardness 
can be calibrated to reflect properties of many materials. The texture is felt as a spatial variation 
in the force provided by the haptic device. The force response - force provided to the operator 
through the haptic device mimics what the user would feel when touching a real object. The 
collision detection and the force feedback calculations (computational algorithms) are important 
elements in the presently claimed invention that are not taught or suggested by Dumoulin or 
Funda. 

(f) With regard to claim 5, the "synchronization" process referred to in the Action is of a 
very different nature than the synchronization process recited in claim 5 and described in the 
instant specification, which refers to time synchronization required between the graphics 
environment that is updated at 30 Hz and the haptics environment that is updated at 1,000 Hz. 
Both the haptics and graphics scene components need to be updated (e.g., based on the objects' 
physical properties and virtual interactions and manipulations) to simulate a dynamic virtual 
environment. These updates are independently performed by the graphics and haptics rendering 
mechanisms. The differences in the rendering rates of graphics (e.g., 30 Hz and 60 for stereo) and 
haptics (1 KHz) require synchronization. Without this synchronization, the haptics and graphics 
scene components would become unstable and appear disconnected. In the preferred 
implementation of the presently claimed invention, the synchronization of the scene components 
is achieved via shared memory, which necessitates that write/read accesses for updating and 
presenting the scene components be mutually exclusive, since reading locations in the memory 
that are simultaneously being updated can result in corrupted values. This exclusivity 
requirement is enforced using memory locking mechanisms. Each process must lock the shared 
memory prior to accessing it and then unlock it when finished. This locking/unlocking mechanism 
must be controlled to avoid "deadlock" situations, where the different processes lock each other 
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out of the memory and infinitely wait for the other process to unlock it. None of these 
mechanisms are disclosed by Dumoulin or Funda. 

(g) With regard to claim 6, Applicants agree with the assertion in the Action that generating 
left eye and right eye images and displaying them to view the stereoscopic image based 3-D 
environment is disclosed by Demoulin and Funda, however, coupling of images based virtual 
environment s to a haptic scene graph and then rendering/displaying the forces as recited in claim 
6, is not taught or suggested in the cited art. As described above, the haptic rendering and 
synchronization of the presently claimed invention is a completely different process, one that is 
nonobvious and nontrivial in nature. 

In light of at least the foregoing, Applicants respectfully submit that claims 2-6 are patent 
over the cited references, as respectfully request reconsideration and withdrawal of all grounds 
for rejecting the claims and objections to the specification. Applicants earnestly solicit a notice 
regarding the allowability of the claims. 

If questions remain, the Examiner is cordially invited to call Applicants' attorney, collect, 
at the number given above. 



Respectfully submitted, 



BHARTI TEMKIN, et al., 
Applicants 



Date: August 5, 2005 




By: 



^Uhn A. Hamilton, Reg. No. 48,946 
Attorney for Applicants 
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